Method and apparatus for determining a three-dimensional position and pose of a fiducial marker

ABSTRACT

Apparatuses and methods train a model and then use the trained model to determine a global three dimensional (3D) position and orientation of a fiduciary marker. In the context of an apparatus for training a model, a wider field-of-view sensor is configured to acquire a static image of a space in which the fiducial marker is disposed and a narrower field-of-view sensor is configured to acquire a plurality of images of at least a portion of the fiducial marker. The apparatus also includes a pan-tilt unit configured to controllably alter pan and tilt angles of the narrower field-of-view sensor during image acquisition. The apparatus further includes a control system configured to determine a transformation of position and orientation information determined from the images acquired by the narrower field-of-view sensor to a coordinate system for the space for which the static image is acquired by the wider field-of-view sensor.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims benefit of U.S. Provisional Application No.63/138,931, filed Jan. 19, 2021, the entire contents of which areincorporated herein by reference.

TECHNOLOGICAL FIELD

An example embodiment relates generally determining a three-dimensionalposition and orientation of a fiducial marker and, more particularly, todetermining the three-dimensional position and orientation of a fiducialmarker utilizing a wider field-of-view sensor as well as a narrowerfield-of-view sensor supported by a pan-tilt unit.

BACKGROUND

A number of applications are dependent upon the determination of theposition of a fiducial marker. However, the position of a fiducialmarker alone may not be sufficient and, instead, six degree of freedom(DOF) pose information, that is, information defining thethree-dimensional position and orientation, of the fiducial marker mustbe determined to locate and interact with the fiducial marker withsufficient precision. In this regard, the three-dimensional position andorientation may be defined in terms of x, y and z coordinates for thethree-dimensional position and pitch, roll and yaw for the orientation.

For example, a fiducial marker may need to be identified, such as interms of six DOF pose information, in conjunction with variousmanufacturing operations, such as manufacturing operations to beperformed in an automated or robotic manner. For example, automatedpainting operations, drilling operations, cutting operations, finishingoperations and other manufacturing operations frequently require theprecise determination of the three-dimensional position and orientationof the various tools utilized by a robot. As such, a fiducial marker maybe attached to the robot manipulator which engages the various tools. Byprecisely identifying the fiducial marker in terms of itsthree-dimensional position and orientation, the position and orientationof the robot manipulator and, in turn, the tools utilized by the robotmay be determined, thereby allowing the manufacturing operations may beperformed in precise positions. Further, movement required in relationto the performance of the manufacturing operations may be preciselyperformed utilizing closed loop control based upon the six DOF poseinformation for the fiducial marker.

Metrology techniques utilized to determine the six DOF pose informationfor a fiducial marker may require relatively expensive equipment, suchas one or more laser range finders, projectors, etc. This equipment isgenerally not only expensive, but may be appropriate for only a limitednumber of tasks and oftentimes must be manually calibrated, therebyincreasing both the time required to identify a fiducial marker and thetraining or experience required of a technician in order to calibratethe specialized equipment. Additionally, at least some of the equipment,such as the sensors, utilized by metrology techniques to determine thesix DOF pose information of a fiducial marker must remain fixed inposition following calibration. In this regard, a plurality of sensors,that is, a sensor wall, may be configured to obtain images of differentportions of a space in which the fiducial marker is disposed. Thisconstraint limits the utility of at least some of the equipment,particularly in instances in which a plurality of sensors are utilizedin combination, since movement of the equipment following calibrationwill require that the calibration process be repeated, thereby extendingthe time required to identify a fiducial marker, such as in terms of thesix DOF pose information.

Additionally, visual metrology, such as utilized in conjunction with theidentification of a fiducial marker for manufacturing operations,generally requires a relatively high level of accuracy. As such,metrology techniques developed for other applications, such as forwide-area surveillance applications, that require less accuracy may beincapable of determining the six DOF pose information of a fiducialmarker with the accuracy demanded by at least some applications, such asthose involving manufacturing operations.

BRIEF SUMMARY

An apparatus and method are provided for training a model to determine athree dimensional (3D) position and orientation of a fiduciary marker ina global coordinate system. The method and apparatus of this exampleembodiment are configured to train the model in such a manner that the3D position and orientation of the fiduciary marker may be determinedwith relatively high accuracy and in an efficient manner, such as inreal time, in at least some embodiments. Further, the method andapparatus of this example embodiment may be implemented utilizingcommercial sensors and may permit repositioning of the sensors followingcalibration, at least about pan and tilt axes, without a requirement foradditional calibration, thereby increasing the ease with which afiducial marker may be identified and its global 3D position andorientation accurately measured. In another example embodiment, a methodand apparatus are provided for determining the 3D position andorientation of a fiducial marker by utilizing, for example, such atrained model.

In an example embodiment, an apparatus for training a model to determinea three-dimensional (3D) position and orientation of a fiducial markeris provided. The apparatus includes a wider field-of-view sensorconfigured to acquire a static image of a space in which the fiducialmarker is disposed. The apparatus also includes a narrower field-of-viewsensor configured to acquire a plurality of images of at least a portionof the fiducial marker. The apparatus further includes a pan-tilt unitconfigured to support the narrower field-of-view sensor and tocontrollably alter a pan angle and a tilt angle at which the narrowerfield-of-view sensor is positioned relative to the fiducial marker suchthat the narrower field-of-view sensor is configured to acquire imagesof at least a portion of the fiducial marker at different pan and tiltangles. The apparatus also includes a control system configured todetermine a transformation of position and orientation informationdetermined from the images acquired by the narrower field-of-view sensorto a coordinate system for the space for which the static image isacquired by the wider field-of-view sensor. In an example embodiment,the pan-tilt unit is configured to alter the pan and tilt angles atwhich the narrower field-of-view sensor is positioned so as to be ableto capture images with the narrower field-of-view sensor throughout theentire sensor view of the wider field-of-view sensor.

The control system of an example embodiment is configured to determinethe transformation by performing a training process. The trainingprocess includes collecting the plurality of images from the narrowerfield-of-view sensor, processing the images in which the fiducial markeris identified in each image and using the model to determine atransformation matrix for the transformation of the position andorientation information determined from the images acquired by thenarrower field-of-view sensor to the coordinate system for the space inwhich the static image is acquired by the wider field-of-view sensor. Inan example embodiment, the wider field-of-view sensor is configured toacquire a second static image of the space in which the fiducial markeris disposed after the fiducial marker has been repositioned. In thisexample embodiment, the narrower field-of-view sensor is configured toacquire a second plurality of images of at least a portion of thefiducial marker at different combinations of pan and tilt angles afterthe fiducial marker has been repositioned. The control system of thisexample embodiment is further configured to determine the transformationof position and orientation information determined from the secondplurality of images acquired by the narrower field-of-view sensor to thecoordinate system for the space for which the static images are acquiredby the wider field-of-view sensor.

The control system of an example embodiment is configured to determinethe transformation by rotating the position and orientation of thefiducial marker that is determined in a local coordinate system of thenarrower field-of-view sensor about a pan axis and a tilt axis definedby the pan-tilt unit. The control system of this example embodiment maybe further configured to determine the transformation by translating theposition and orientation of the fiducial marker to determine into thelocal coordinate system of the narrower field-of-view sensor from anorigin to the pan axis and the tilt axis defined by the pan-tilt unit.In this example embodiment, the control system may be further configuredto determine the transformation by transforming from the localcoordinate system at pan and tilt angles of 0° to the coordinate systemfor the space for which the static image is acquired by the widerfield-of-view sensor. In an example embodiment, the control system isconfigured to determine the transformation by determining modelparameters to minimize a loss function defining an error metric betweena position and orientation of the fiducial marker in the coordinatesystem for the space for which the static image is acquired by the widerfield-of-view sensor and in a local coordinate system of the narrowerfield-of-view sensor. The control system of an example embodiment isconfigured to determine the model parameters by evaluating an errormetric between estimates of position and orientation from the widerfield-of-view sensor and from the narrower field-of-view sensor usingrespective positions of 3D features of one or more fiducial markers inrespective positions and orientations.

In another example embodiment, a method for training a model todetermine a three dimensional position and orientation of a fiducialmarker is provided. The method includes acquiring, with a widerfield-of-view sensor, a static image of the space in which the fiducialmarker is disposed. The method also includes sequentially positioning anarrower field-of-view sensor to have different combinations of a panangle and a tilt angle relative to the fiducial marker. The methodfurther includes acquiring, with the narrower field-of-view sensor, aplurality of images of at least a portion of the fiducial marker at adifferent pan and tilt angles. The method also includes determining atransformation of position and orientation information determined fromthe images acquired by the narrower field-of-view sensor to a coordinatesystem for the space for which the static image is acquired by the widerfield-of-view sensor.

The method of an example embodiment determines the transformation byperforming a training process. In this regard, the training process isperformed by collecting the plurality of images from the narrowerfield-of-view sensor, processing the images in which the fiducial markeris identified in each image and using the model to determine atransformation matrix for transformation of the position and orientationinformation determined from the images acquired by the narrowerfield-of-view sensor to the coordinate system for the space for whichthe static image is acquired by the wider field-of-view sensor. In anexample embodiment, the method also includes acquiring, with the widerfield-of-view sensor, a second static image of the space in which thefiducial marker is disposed after the fiducial marker has beenrepositioned. The method of this example embodiment also includesacquiring, with the narrower field-of-view sensor, a second plurality ofimages of at least a portion of the fiducial marker at differentcombinations of pan and tilt angles after the fiducial marker has beenrepositioned. The method of this example embodiment further includesdetermining the transformation of position and orientation informationdetermined from the second plurality of images acquired by the narrowerfield-of-view sensor to the coordinate system for the space for whichthe static images are acquired by the wider field-of-view sensor.

The method of an example embodiment determines the transformation byrotating a position and orientation of the fiducial marker determined ina local coordinate system of the narrower field-of-view sensor about apan axis and a tilt axis defined by the pan and tilt unit. The method ofthis example embodiment may also determine the transformation bytranslating the position and orientation of the fiducial markerdetermined in the local coordinate system of the narrower field-of-viewsensor from an origin to the pan axis and the tilt axis defined by thepan and tilt unit. The method of this example embodiment may alsodetermine the transformation by transforming from the local coordinatesystem at pan and tilt angles of 0° to the coordinate system for thespace for which the static image is acquired by the wider field-of-viewsensor. In an example embodiment, the method determines thetransformation by determining model parameters to minimize a lossfunction defining an error metric between a position and orientation ofthe fiducial marker in the coordinate system for the space for which thestatic image is acquired by the wider field-of-view sensor and in thelocal coordinate system of the narrower field-of-view sensor. The methodof an example embodiment determines the model parameters by evaluatingan error metric between estimates of position and orientation from thewider field-of-view sensor and from the narrower field-of-view sensorusing respective positions of 3D features of one or more fiducialmarkers in respective positions and orientations.

In a further example embodiment, an apparatus is provided fordetermining a three dimensional position and orientation of a fiducialmarker. The apparatus includes a wider field-of-view sensor configuredto acquire a static image of a space in which the fiducial marker isdisposed. The apparatus also includes a narrower field-of-view sensorconfigured to acquire an image of at least a portion of the fiducialmarker and a pan-tilt unit configured to support the narrowerfield-of-view sensor and to controllably define a pan angle and a tiltangle at which the narrower field-of-view sensor is positioned relativeto the fiducial marker such that the narrower field-of-view sensor isconfigured to acquire the image of at least a portion of the fiducialmarker. The apparatus further includes a control system configured toutilize a transformation to determine, independent of any positioninformation for the wider field-of-view and narrower field-of-viewsensors, an estimate of the position and orientation of the fiducialmarker in a coordinate system for the space for which the static imageis acquired by the wider field-of-view sensor, based on the imagesacquired from the wider field-of-view and narrower field-of-view sensorsviewing the fiducial marker and also based on the pan and tilt angles atwhich the narrower field-of-view sensor is positioned relative to thefiducial marker upon acquiring the image.

The control system of an example embodiment is further configured todetermine a position and orientation of the fiducial marker in a localcoordinate system of the narrower field-of-view sensor. In this exampleembodiment, the control system is configured to utilize thetransformation to convert the position and orientation of the fiducialmarker in the local coordinate system of the narrower field-of-viewsensor to the position and orientation of the fiducial marker in thecoordinate system for the space for which the static image is acquiredby the wider field-of-view sensor based upon the pan angle and tiltangle of the pan-tilt unit.

In an example embodiment, the narrower field-of-view sensor isconfigured to acquire the image of at least a portion of the fiducialmarker with more pixels representative of the fiducial marker than thestatic image acquired by the wider field-of-view sensor. The controlsystem of an example embodiment is configured to determine the estimateof the position and orientation of the fiducial marker in real time withacquisition of the image of at least the portion of the fiducial markerby the narrower field-of-view sensor. In an example embodiment, thetransformation is based upon model parameters determined to minimize theloss function defining an error metric between the position andorientation of the fiducial marker in the coordinate system for thespace for which the static image is acquired by the wider field-of-viewsensor and in the local coordinate system of the narrower field-of-viewsensor.

In yet another example embodiment, a method if provided for determininga three-dimensional position and orientation of a fiducial marker. Themethod includes acquiring, with a wider field-of-view sensor, a staticimage of a space in which the fiducial marker is disposed. The methodalso includes controllably defining a pan angle and a tilt angle atwhich a narrower field-of-view sensor is positioned relative to thefiducial marker and acquiring, with the narrower field-of-view sensor,an image of at least a portion of the fiducial marker. The methodfurther includes utilizing a transformation to determine, independent ofany position information for the wider field-of-view and narrowerfield-of-view sensors, an estimate of the position and orientation ofthe fiducial marker in a coordinate system for the space for which thestatic image is acquired by the wider field-of-view sensor, based on theimages acquired from the wider field-of-view and narrower field-of-viewsensors viewing the fiducial marker and also based on the pan and tiltangles at which the narrower field-of-view sensor is positioned relativeto the fiducial marker upon acquiring the image.

The method of an example embodiment also includes determining theposition and orientation of the fiducial marker in a local coordinatesystem of the narrower field-of-view sensor. The method of this exampleembodiment utilizes the transformation to convert the position andorientation of the fiducial marker in the local coordinate system of thenarrower field-of-view sensor to the position and orientation of thefiducial marker in coordinate system for the space for which the staticimage is acquired by the wider field-of-view sensor based upon the panangle and the tilt angle.

In an example embodiment, the image of at least a portion of thefiducial marker acquired by the narrower field-of-view sensor includesmore pixels representative of the fiducial marker than the static imageacquired by the wider field-of-view sensor. In an example embodiment,the estimate of the position and orientation of the fiducial marker isdetermined in real time with the acquisition of the image of at least aportion of the fiducial marker by the narrower field-of-view sensor. Thetransformation of an example embodiment is based upon model parametersdetermined to minimize the loss function defining an error metricbetween the position and orientation of the fiducial marker in thecoordinate system for the space for which the static image is acquiredby the wider field-of-view sensor and in the local coordinate system ofthe narrower field-of-view sensor.

BRIEF DESCRIPTION OF THE DRAWINGS

Having thus described certain example embodiments of the presentdisclosure in general terms, reference will hereinafter be made to theaccompanying drawings, which are not necessarily drawn to scale, andwherein:

FIG. 1 is a perspective view of a workpiece and an associated paint headfor which the six degree of freedom (DOF) pose information, namely,position and orientation, is determined in accordance with an exampleembodiment of the present disclosures;

FIG. 2 illustrates an apparatus configured to determine the position andorientation of a fiducial marker in accordance with an exampleembodiment of the present disclosure;

FIG. 3 depicts a pan-tilt unit and a narrower field-of-view sensorsupported by the pan-tilt unit in accordance with an example embodimentof the present disclosure;

FIG. 4 is a flowchart for the operations performed, such as by theapparatus of FIG. 2, in order to train the model in order to determinethe position and orientation of a fiducial marker in accordance in anexample embodiment of the present disclosure;

FIG. 5 illustrates a ChArUco board;

FIG. 6 is a flowchart of the operations performed, such as by theapparatus of FIG. 2, in order to implement a training process inaccordance with an example embodiment of the present disclosure; and

FIG. 7 is a block diagram of the operations performed, such as by theapparatus of FIG. 2, in order to determine the position and orientationof a fiducial marker utilizing a trained model in accordance with anexample embodiment of the present disclosure.

DETAILED DESCRIPTION

The present disclosure now will be described more fully hereinafter withreference to the accompanying drawings, in which some, but not allaspects are shown. Indeed, the disclosure may be embodied in manydifferent forms and should not be construed as limited to the aspectsset forth herein. Rather, these aspects are provided so that thisdisclosure will satisfy applicable legal requirements. Like numbersrefer to like elements throughout.

A method and apparatus are provided for training a model to determine athree dimensional (3D) position and orientation, of a fiducial marker aswell as a method and apparatus for utilizing the trained model todetermine the 3D position and orientation of a fiducial marker. Theaccurate determination of the 3D position and orientation, also known asthe six degree of freedom (DOF) pose information, of a fiducial markermay be utilized in conjunction with a wide variety of applications thatare dependent upon the accurate identification of the fiducial marker.

For example, a number of manufacturing operations are dependent upon theaccurate identification and locating of one or more objects. As shown inFIG. 1 by way of example, but not of limitation, the accuratedetermination of the position and orientation of a paint head 10 carriedby a robotic arm 12 relative to workpiece 14 is useful during paintingoperations such that the workpiece or at least specific portions of theworkpiece are appropriately painted while other portions of theworkpiece remain unpainted and previously painted portions of theworkpiece are not unnecessarily repainted. By accurately identifying theposition and orientation of the paint head 10 relative to the workpiece14, the proper portion of the workpiece may be painted in an efficientmanner, thereby conserving resources during the manufacturing process.In this regard, as a result of the accurate determination of theposition and orientation of the paint head 10 relative to the workpiece14, paint is conserved by uniformly applying a coat of paint of adesired thickness, while not unnecessarily re-painting previouslypainted surfaces, thereby also avoiding undesirable increases in theweight of the aircraft. Further, by accurately determining the positionand orientation of the paint head 10 relative to the workpiece 14 andcorrespondingly ensuring that all portions of the workpiece that areintended to be painted have, in fact, been painted, the resultingappearance of the workpiece may be improved along with the weatherprotection of the aircraft offered by the paint.

Although described above in conjunction with the accurate determinationof the position of a paint head 10 relative to a workpiece 14, themethod and apparatus of an example embodiment may also be utilized toaccurately determine the position and orientation of any of a variety ofobjects in conjunction with other manufacturing operations includingdrilling operations, cutting operations, etc. Further, the method andapparatus of an example embodiment may be utilized in conjunction withthe accurate determination of the position and orientation of an endeffector, a robotic arm or an object, such as the 6 DOF poseinformation, in applications other than manufacturing.

The apparatus 20 of an example embodiment is depicted in FIG. 2. Theapparatus 20 includes one or more wider field-of-view sensors 22 and oneor more narrower field-of-view sensors 24. Although different types ofsensors may be utilized in order to acquire images, the sensors of anexample embodiment are cameras configured to acquire images of therespective fields of view. The wider field-of-view sensor 22 has a widerfield-of-view than the narrower field-of-view sensor 24. The widerfield-of-view sensor 22 has a shorter focal length than the narrowerfield-of-view sensor 24. Additionally, while the wider field-of-viewsensor 22 and the narrower field-of-view of sensor 24 may have the sameresolution, the narrower field-of-view sensor of one example embodimenthas a greater resolution than the resolution of the wider field-of-viewsensor. The wider field-of-view sensor 22 and the narrower field-of-viewsensor 24 also generally disallow autofocus and have fixed zoom settingssuch that neither the focus distance nor the zoom setting changes.

The wider field-of-view sensor 22 may be fixed in position relative tothe space in which the fiducial marker will be located. In this regard,the wider field-of-view sensor 22 is positioned such that the entirespace in which the fiducial marker could potentially be located isacquired within the same image. For example, in an instance in which thefiducial marker could be positioned at any position upon a workpiece,the wider field-of-view sensor 22 is positioned such that a static imagethat is acquired includes the entire workpiece.

The narrower field-of-view sensor 24 is also generally positioned at afixed position, such as a position having fixed x, y and z coordinates,although the narrower field-of-view sensor of another example embodimentmay be configured to be controllably repositioned, such as by mountingthe narrower field-of-view sensor upon a rail that facilitatestranslation of the narrower field-of-view sensor along a predefined pathdefined by the rail. As described below, however, the narrowerfield-of-view sensor 24 is configured to rotate about a pan axis and atilt axis. The narrower field-of-view sensor 24 is configured to acquirean image of a portion of the same space that is acquired by the staticimage of the wider field-of-view sensor 22. However, the narrowerfield-of-view sensor 24 generally does not acquire an image thatincludes the entire space, but instead only that portion of the space inwhich the fiducial marker is located.

As shown in FIG. 2, the apparatus 20 of an example embodiment alsoincludes a pan axis tilt unit 26. The pan-tilt unit 26 is configured tosupport the narrower field-of-view sensor 24 and to controllably andseparately alter a pan angle and a tilt angle at which the narrowerfield-of-view sensor is positioned relative to the space and, moreparticularly, relative to the fiducial marker located within the space.As such, the narrower field-of-view sensor 24 is configured to acquireimages of a portion of the space, such as the portion of the spacewithin which the fiducial marker is disposed, at different combinationsof pan and tilt angles. As the fiducial marker may be positioned at anyof various positions throughout the space, the pan-tilt unit 26 isconfigured to controllably reposition the narrower field-of-view sensor24 in terms of the pan and tilt angles such that the narrowerfield-of-view sensor is capable of viewing the entire space within thestatic image captured by the wider field-of-view sensor 22, even thoughthe narrower field-of-view sensor is only capable of viewing a portionof the entire space at any particular combination of pan and tiltangles.

Although the pan-tilt unit 26 may be configured in different manners,the pan-tilt unit of an example embodiment is depicted in FIG. 3 toinclude a platform 30 and an associated tilt servo motor 32 that isconfigured to controllably alter the tilt angle of the platform and, inturn, the tilt angle of the narrower field-of-view sensor 24.Additionally, the pan-tilt unit 26 of this example embodiment mayinclude a pan servo motor 34 configured to controllably alter the panangle of the platform 30 and, in some embodiments, both the platform andthe tilt servo motor 32. By altering the pan angle of the platform 30,the pan servo motor 34 also controllably alters the pan angle of thenarrower field-of-view sensor 24.

The apparatus 20 of FIG. 2 also includes a control system 28. Thecontrol system 28 may be embodied in a variety of different mannersincluding by a controller, a processor and any of a variety of computingdevices, such as a personal computer, a computer workstation, a serveror the like. In an example embodiment, the control system 28 isconfigured to determine a transformation of position and orientationinformation determined from the images acquired by the narrowerfield-of-view sensor 24 to a coordinate system, that is, a worldcoordinate system, for the space from which the static image is acquiredby the wider field-of-view sensor 22. In this regard, the transformationmay be at least partially defined by the model having a plurality ofmodel parameters that serve to translate the position and orientation ofthe fiducial marker in a local coordinate system defined by the narrowerfield-of-view sensor 24 to the world coordinate system of the widerfield-of-view sensor 22.

In another example embodiment, the control system 28 is configured toutilize the transformation, such as defined by a trained model, todetermine an estimate of the position and orientation of a fiducialmarker in the coordinate system for the space from which the staticimage is acquired by the wider field-of-view sensor 22. Thisdetermination of the estimate of the position and orientation of afiduciary marker may be based on the images acquired from the widerfield-of-view and narrower field-of-view sensors 22, 24 and also basedon the pan and tilt angles at which the narrower field-of-view sensor ispositioned relative to the fiducial marker upon acquiring the image.Thus, the control system 28 of this example embodiment is configured todetermine the position and orientation of a fiducial marker in the worldcoordinate system in an efficient and reliable manner, such as in realtime with a relatively small error, based on an image of a fiducialmarker captured by the narrower field-of-view sensor 24 positioned at aparticular combination of pan and tilt angles.

Referring now to FIG. 4, the operations performed, such as by theapparatus 20 of FIG. 2, in order to train a model to determine the 3Dposition and orientation of a fiducial marker are depicted. The fiducialmarker may be located coincident with an object for which the positionand orientation is to be determined. As shown in FIG. 1, for example,the fiducial marker 16 may be placed on and carried by an object to beidentified, such as by being carried by a paint head 10 in order todetermine the position and orientation of the paint head relative to aworkpiece 14.

Various types of fiducial markers may be utilized including a ChArUcoboard. One example of a ChArUco board 54 is depicted in FIG. 5. AChArUco board includes a ChArUco pattern which is a combination of acheckerboard 56 and a grid of ArUco markers 58. The checkerboardportions provide the structure required for the calibration anddetection in order to determine position and orientation, while theArUco markers identify specific sections of the ChArUco pattern. TheArUco markers therefore permit the ChArUco pattern to be utilized ininstances in which only a partial or an occluded view of the ChArUcoboard is available since the ArUco markers permit the portion of theChArUco board that is visible to be identified. While a ChArUco boardmay be utilized as a fiducial marker, other types of fiducial patternsmay be utilized in other example embodiments.

After having positioned the fiducial marker, such as upon an object forwhich the position and orientation is to be determined, and as shown inblock 40 of FIG. 4, the wider field-of-view sensor 22 is configured toacquire a static image of the space in which the fiducial marker isdisposed including, for example, the entire space in which the fiducialmarker could potentially be disposed. In an example embodiment in whichthe fiducial marker could be located anywhere upon a workpiece, thewider field-of-view sensor 22 is configured to acquire static image of aspace that includes the entire workpiece, thereby necessarily includingthe position at which the fiducial marker will be disposed. In anexample embodiment, the fiducial marker is located perpendicularly ornearly perpendicularly to the line-of-sight of the wider field-of-viewsensor 22 to facilitate the accuracy with which the position andorientation of the fiducial marker is determined in the world coordinatesystem defined by the wider field-of-view sensor.

As shown in block 42 for a fiducial marker located at a first position,the pan-tilt unit 26 is configured to sequentially position the narrowerfield-of-view sensor 24 at different combinations of pan angle and tiltangle relative to the fiduciary marker. Although the pan-tilt unit 26may be configured to sequentially position the narrower field-of-viewsensor 24 to have any number of different combinations of the pan andtilt angles relative to the fiducial marker, the pan-tilt unit of oneexample embodiment is configured to sequentially position the narrowerfield-of-view sensor to have the plurality of different pan angles and aplurality of different tilt angles. Although the number of pan anglesand the number of tilt angles may be different, the pan-tilt unit 26 ofone embodiment is configured to sequentially position the narrowerfield-of-view sensor 24 to have the same number of different pan anglesand different tilt angles, such as 16 different pan angles and 16different tilt angles. Although the step size or angular incrementbetween pan angles and tilt angles may be different, the pan-tilt unit26 of an example embodiment is configured to alter the pan angle and thetilt angle such that the step size or angular increment between adjacentpan angles and adjacent tilt angles is equal.

Regardless of the number of different pan angles and tilt angles, thepan-tilt unit 26 is configured to determine the pan angles and the tiltangles such that at least a portion of the fiducial marker and, in oneembodiment, at least half of the fiducial marker, is within the imageacquired by the narrower field-of-view sensor 24. In an embodiment inwhich the fiducial marker has a ChArUco pattern, the pan-tilt unit 26 isconfigured to alter the pan angle to range between a smallest pan angleand a largest pan angle such that at the smallest pan angleapproximately half of the ChArUco pattern is visible at the left side ofthe image captured by the narrower field-of-view sensor 24, while at thelargest pan angle approximately half of the ChArUco pattern is visibleat a right side of the image captured by the narrower field-of-viewsensor. Similarly, the pan-tilt unit 26 of this example embodiment maybe configured to alter the tilt angle to range between a smallest tiltangle and a largest tilt angle such that at the smallest tilt angleapproximately half of the ChArUco pattern is visible at the bottom ofthe image captured by the narrower field-of-view sensor 24, while at thelargest tilt angle approximately half of the ChArUco pattern is visibleat the top of the image captured by the narrower field-of-view sensor.

As shown in block 44 of FIG. 4, the narrower field-of-view sensor 24 isconfigured to acquire a plurality of images of the fiducial marker atthe different combinations for pan and tilt angles. Thus, the narrowerfield-of-view sensor 24 of an example embodiment is configured toacquire an image of the fiducial marker at each different combination ofpan and tilt angles defined by the pan-tilt unit 26.

As shown in block 46 of FIG. 4, the control system 28 is configured todetermine a transformation of the position and orientation informationdetermined from the images acquired from the narrower field-of-viewsensor 24 to a coordinate system, that is, a world coordinate system,for the space that is the subject of the static image acquired by thewider field-of-view sensor 22. The control system 28 of an exampleembodiment is configured to determine the transformation by performing atraining process. As shown in FIG. 6, for example, the training processimplemented by the control system 28 may be configured to collect aplurality of images from the narrower field-of-view sensor 24 andprocess the images in which the fiducial marker is identified in eachimage to determine the position and orientation information. See blocks60 and 62. The control system 28 of this example embodiment is alsoconfigured to implement the training process by using the model todetermine a transformation matrix for the transformation of the positionand orientation information determined from the images acquired by thenarrower field-of-view sensor 24 to the coordinate system for the spacefor which the static image is acquired by the wider field-of-view sensor22. See block 64 of FIG. 6.

During this training phase, the position and orientation informationdetermined from the images acquired by the narrower field-of-view sensor24 is transformed into 6 DOF pose information expressed in the worldcoordinate system as determined from the static image captured by thewider field-of-view sensor 22. The 6 DOF pose information of thefiducial marker may be denoted P_(World) and, in one embodiment, isrepresented by a matrix, such as a 4×4 matrix having a block, such asthe top left 3×3 block, which is a rotation matrix representing theorientation of the fiducial marker and one or more other entries, suchas the top three entries of the right most column of the 4×4 matrix,that represent the position of the fiducial marker, and the bottom rowof the 4×4 matrix having all zeros (0.0) except for the last column ofthe bottom row which is 1.0.

Within each image acquired by the narrower field-of-view sensor 24 at arespective combination of pan and tilt angles, the 6 DOF poseinformation, that is, the position and orientation, of the fiducialmarker in the local coordinate system defined by the narrowerfield-of-view sensor is extracted and is denoted as P_(NFOV). In anexample in which the fiducial marker includes a ChArUco pattern, thecontrol system 28 is configured to extract the 6 DOF pose informationfrom the image captured by the narrower field-of-view sensor 24 by firstdetecting the ArUco features and then refining those features to thecheckerboard features between them prior to solving theProspective-n-Point problem using a 3D model of the fiducial marker. Theresult of this extraction is six DOF pose information for the fiducialmarket in the local coordinate system defined by the narrowerfield-of-view sensor 24.

As noted above, after the position and orientation information isdetermined from the images acquired by the narrower field-of-view sensor24, the control system 28 transforms the position and orientationinformation to the 6 DOF pose information in the world coordinatesystem. In relation to this transformation, a plurality of dynamicextrinsic parameters are unknown, but the intrinsic parameters of thewider field-of-view sensor 22 and the narrower field-of-view sensor 24are known, e.g., predefined, including a representation of their focallength and distortion properties, among others. In order to effect thetransform while working with a plurality of unknown dynamic extrinsicparameters, the control system 28 of an example embodiment is configuredto implement a series of transforms.

In this regard, since the local coordinate system of the widerfield-of-view sensor 22 is utilized as the world coordinate system,there is no transformation necessary from pose information P_(WFOV)(namely, position and orientation) expressed in the local coordinatesystem of the wider field-of-view sensor and pose information P_(World)expressed in the world coordinate system such that P_(World)=P_(WFOV).In contrast, in order to express pose information P_(NFOV) (position andorientation) from the local coordinate system of a narrowerfield-of-view sensor 24 in the world coordinate system of the widerfield-of-view sensor 22, the control system 28 is configured totransform the pose information by a series of rotations and translationsincluding the rotations about the pan and tilt axes defined by thepan-tilt unit 26.

The control system 28 of an example embodiment is configured todetermine the transformation by rotating the position and orientation ofthe fiducial marker that is determined in the local coordinate system ofthe narrower field-of-view sensor 24 about a pan axis and a tilt axisdefined by the pan-tilt unit 26. The control system 28 of this exampleembodiment is also configured to determine the transformation bytranslating the position and orientation of the fiducial marker that isdetermined in the local coordinate system of the narrower field ofsensor 24 from an origin of this local coordinate system to the pan axisand to the tilt axis as defined by the pan-tilt unit 26. The controlsystem of this example embodiment is also configured to determine thetransformation by transforming from the local coordinate system at panand tilt angles of 0° to the coordinate system for the space for whichthe static image is acquired by the wider field-of-view sensor 22, thatis, the world coordinate system.

This transformation may be expressed in a different manner based uponthe plurality of different transforms required to convert from the localcoordinate system of the narrower field-of-view sensor 24 to the worldcoordinate system of the wider field-of-view sensor 22. In relation tothe example embodiment of the apparatus 20 depicted in FIG. 2, a firsttransformation is defined between the narrower field-of-view sensor 24and the tilt coordinate system. This first transformation includes arotation R_(NFOV→Tilt) and a translation T_(NFOV→Tilt) from the narrowerfield-of-view sensor 24 to the tilt coordinate system. This firsttransformation from the narrower field-of-view sensor 24 to the tiltcoordinate system includes 3 parameters associated with rotation,namely, yaw, pitch and roll, in the tilt coordinate system and 3translation parameters, namely x, y and z, associated with translationto the tilt coordinate system. Each of these 6 parameters for thistransformation is unknown. A second transformation is defined betweenthe tilt coordinate system and the pan coordinate system including arotation R_(Tilt→Pan) in terms of pitch and a translation T_(Tilt→Pan)in terms of x, y and z. Of these four parameters, the tilt angle, whichcorresponds to the pitch, is known, but the 3 translation parameters areunknown.

The transformation also includes a third transformation from the pancoordinate system to the base, that is, the support structure, e.g.,floor, table, etc., for the apparatus 20. The third transformationincludes a rotation R_(Pan→Base) in terms of yaw and a translationT_(Pan→Base). In some embodiments, the origins of the coordinate systemsof the base and the pan-tilt unit 26 are coincident such that there isno translation between the pan coordinate system and the base, therebyleaving only one parameter, yaw, to be determined, with the translationT_(Pan→Base) being omitted. As the pan angle corresponds to yaw, the yawparameter is known and the third transformation introduces no unknownparameters. Further, a fourth transformation from the base to the widerfield-of-view sensor 22 includes a rotation R_(Base→World) and atranslation T_(Base→World). This fourth transformation from the base tothe wider field-of-view sensor 22 includes 3 parameters associated withrotation, namely, yaw, pitch and roll, and 3 translation parameters,namely x, y and z. Each of these 6 parameters for this transformation isstatic and is unknown. In this regard, the 6 parameters are staticbecause the parameters do not change when the pan and tilt angles of thepan-tilt unit 26 change in the system.

As such, the overall transformation from the narrower field-of-viewsensor 24 to the wider field-of-view sensor 22 includes 17 parameters ofwhich two, that is, the pan and tilt angles, are known such that 15parameters remain unknown, Taking into account these individualtransformations, the overall transformation from the local coordinatesystem of the narrower field-of-view sensor 24 to the world coordinatesystem of the wider field-of-view senor 22 may be expressed as:

P_(World) = T_(Base → WFOV)R_(Base → WFOV)R_(Pan → Base)T_(Tilt → Pan)R_(Tilt → Pan)T_(NFOV → Tilt)R_(NFOV → Tilt)P_(NFOV)

wherein R_(Base→WFOV) T_(Base→WFOV) are arbitrary rigid transformationsrepresenting the position and orientation of the pan-tilt unit 26 in thewider field-of-view or world coordinates. In an example embodiment, eachof the rotation R and translation T transformations above can berepresented as a 4×4 matrix, respectively, where the top left 3×3 blockof R is filled with a 3×3 rotation matrix, and the remaining entries ofR filled with 0.0 except for the last row of the last column entry whichis filled with 1.0, and where T is a 4×4 identity matrix except for thetop three entries of the right most column, which are filled with thetranslation components in x, y, and z, respectively.

The transformation determined by the control system of an exampleembodiment may be expressed as:

P_(World) = E(pan, tilt; ω)P_(NFOV)

wherein E(pan, tilt; ω) is a matrix representation, such as a 4×4 matrixrepresentation, of the transformation from the local coordinate systemof the narrower field-of-view sensor 24 to the world coordinate systemof the wider field-of-view sensor 22 for particular angles of pan andtilt, obtained by combining the sequence of transformations using matrixmultiplication operations. The vector ω represents the model parametersof the transformation, which will depend on the parameterization. Inthis regard, the model parameters ω are the union of the staticparameters ω_(s) and the dynamic parameters ω_(d) of the model. Asdescribed in relation to the foregoing equation, R_(Base→WFOV)T_(Base→WFOV) are static parameters of the model, while R_(Pan→Base)T_(Pan→Base), R_(Tilt→Pan) T_(Tilt→Pan) and R_(NFOV→Tilt) T_(NFOV→Tilt)are dynamic parameters of the model. According to this parameterization,there are 15 (unknown) parameters of which 6 are static and 9 aredynamic.

The control system 28 of an example embodiment is configured todetermine the transformation by determining model parameters to minimizea loss function defining an error metric between a position andorientation of the fiducial marker in a coordinate system, e.g., theworld coordinate system, for the space related to the static image thatis acquired by the wider field-of-view sensor 22 and a local coordinatesystem of the narrower field-of-view sensor 24. See block 48 of FIG. 4.In this regard, the control system 28 is configured to determine thetransformation by calibrating the model to repeatedly minimize thedisagreement between various six DOF pose estimates of the fiducialmarker from the world coordinate system of the wider field-of-viewsensor 22 and those from the local coordinate system of the narrowerfield-of-view sensor 24, by varying some or all of the model parameters.For example, the control system 28 may be configured to determine themodel parameters by evaluating an error metric between pose estimatesfrom the wider field-of-view sensor 22 and from the narrowerfield-of-view sensor 24 using respective positions of 3D features of oneor more fiducial markers in respective poses. In an example embodiment,the error metric that is utilized between poses is an error metricbetween a point cloud consisting of the predicted fiducial featurepositions in 3D, where the 3D feature locations can be derived by thefiducial marker design. In an example embodiment in which the fiducialmarker is a ChaRuCo pattern, the chessboard corners of the ChaRuCopattern can be used as the fiducial features, and their 3D locations canbe derived from the pose of the fiducial marker. In other words, for twodifferent poses P_(X) and P_(Y), X₁, X₂, . . . , X_(n) are the 3Dfeature positions for the first pose and Y₁, Y₂, . . . Y_(n) are the 3Dfeature positions for the second pose, where Xi and Yi, i=1˜n, arevectors of 3D locations of the features. In an example embodiment, theerror metric L utilized by the control system 28 is defined by the meanabsolute error (MAE) as follows:

${L_{MAE}( {P_{X},P_{Y}} )} = {\frac{1}{n}{\sum\limits_{i}^{\;}{{X_{i} - Y_{i}}}}}$

In an alternative embodiment, the error metric L utilized by the controlsystem 28 is defined as the root-mean-squared error (RMSE) as follows:

${L_{RMSE}( {P_{X},P_{Y}} )} = \sqrt{\frac{1}{n}{\sum\limits_{i}^{\;}{{X_{i} - Y_{i}}}^{2}}}$

Once the error metric has been defined, the relationship between theposition and orientation information in the world coordinate system andthe transformation of the position and orientation from the localcoordinate system may be defined as:

P_(WFOV) = E(pan, tilt; ω)P_(NFOV)

The control system 28 of this example embodiment is then configured tooptimize, such as by minimizing, a loss function, such as L_(MAE) orL_(RMSE), which describes how far the foregoing equation is from beingsatisfied, averaged across all of the measurements of the fiducialmarker. In this regard in which L defines the error metric between thesix DOF pose information of the fiducial marker, the control system 28of an example embodiment is configured to calibrate the model by findingall of the model parameters ω which minimize the following loss functionin a single step:

$\sum\limits_{P_{World}}^{\;}{\frac{1}{m_{P_{World}}}{\sum\limits_{P_{WFOV},P_{NFOV}}^{\;}{L( {P_{WFOV},{{E( {{pan},{{tilt};\omega}} )}P_{NFOV}}} )}}}$

wherein m_(P) _(World) represents the number of images acquired by thewider field-of-view sensor 22 and the narrower field-of-view sensor 24,that is, P_(WFOV) and P_(NFOV), taken under different combinations ofpan and tilt angles while the fiducial marker was in the pose P_(World).The control system 28 may be configured to perform this minimization invarious manners including in accordance with Levenberg-Marquardtoptimization or with other nonlinear optimization algorithms such as theBroyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm.

In an alternative embodiment, the control system 28 is configured as atwo-step process. In a first step, the control system 28 is configuredto solve for the model parameters by treating the poses of the fiduciarymarker as parameters in the optimization step and then fitting themsimultaneously while fitting the model parameters ω. In other words,P_(world) is utilized n place of P_(WFOV) and the control system thensolves for P_(World) while concurrently solving for the model parametersco. In particular, the control system 28 of this example embodiment isconfigured to first determine or fit the dynamic parameters ω_(d), thatis, the subset of the model parameters ω that are dynamic, and thefiducial poses (P_(World)) by optimizing, e.g., minimizing:

$\sum\limits_{P_{World}}^{\;}{\frac{1}{m_{P_{World}}}{\sum\limits_{P_{WFOV},P_{NFOV}}^{\;}{L( {P_{World},{{E_{d}( {{pan},{{tilt};\omega_{d}}} )}P_{NFOV}}} )}}}$

with respect to the dynamic parameters ω_(d) of the model and to thefiducial poses (P_(World)). E_(d)(pan, tilt; ω_(d)) is a 4×4 matrixrepresenting the transformation of the dynamic model in which pan andtilt are known from the data collection and ω_(d) are the correspondingdynamic model parameters that are to be determined. While determiningthe dynamic parameters ω_(d) and the fiducial poses (P World), thestatic model parameters ω_(s) may be set to a set of fixed values, suchas 0 translations and 0 rotations. The dynamic parameters achieved inthis step are denoted as ω_(d).

In the second step, the control system 28 of this example embodiment isthen configured to determine or fit the static parameters ω_(s) of themodel while the dynamic parameters ω_(d) are fixed to the previouslydetermined values and foregoing equation is minimized while allowingonly the static parameters to change as shown below:

$\sum\limits_{P_{World}}^{\;}{\frac{1}{m_{P_{World}}}{\sum\limits_{P_{WFOV},P_{NFOV}}^{\;}{L( {P_{WFOV},{{E( {{pan},{{tilt};{\overset{\sim}{\omega}}_{d};\omega_{s}}} )}P_{NFOV}}} )}}}$

wherein E is a 4×4 matrix representing the transformation of the modelas before except that the dynamic model parameters associated with{tilde over (ω)}_(d) that were determined in the first step are fixed,and only the static model parameters ω_(s) are allowed to change duringthe minimization. The control system 28 of this example embodimenttherefore effectively decomposes the calibration of the full network ofthe wider field-of-view sensor 22 and the pan-tilt mounted narrowerfield-of-view sensors 24 into the separate calibration of the pan-tiltmounted narrower field-of-view sensors and a subsequent calibrationunifying all of the separate calibrations into a single coordinatesystem.

In an example embodiment, the control system 28 is also configured todetermine a measure of the error, that is, the reprojection errorbetween: (i) the transformation of the position and orientationinformation acquired in the local coordinate system by the narrowerfield-of-view sensor 24 to the world coordinate system of the widerfield-of-view sensor 22 and (ii) the actual position and orientation ofthe fiducial marker as determined from the static image acquired by thewider field-of-view sensor. The control system 28 of this exampleembodiment is configured to analyze the measure of the error todetermine whether the resulting error indicates that the model,including the model parameters, has converged. See block 50 of FIG. 4.In an instance in which the error satisfies a predefined errorthreshold, such as by being less than the predefined error threshold,the control system 28 of this example embodiment is configured todetermine that the model has converged and the trained model is ready tobe utilized to transform position and orientation information acquiredby the narrower field-of-view camera 24 to the coordinate system of thewider field-of-view sensor 22. Conversely, in an instance in which theerror is determined not to have converged, such as in an instance inwhich the control system 28 determines that the error fails to satisfy,such as by being greater than, a predefined error threshold, thefiducial marker is repositioned as shown in block 52, and this processis repeated in order to refine the model including the model parameters.

In this example embodiment, after the fiducial marker have beenrepositioned, such as relative to a workpiece, the wider field-of-viewsensor 22 acquires a second static image of the space in which thefiducial marker is disposed and the narrower field-of-view sensor 24acquires a plurality of images of at least a portion of the fiducialmarker at different combinations of pan and tilt angles. The controlsystem 28 of this example embodiment is then configured to determine thetransformation of position and orientation information determined fromthe second plurality of images acquired by the narrower field-of-viewsensor 24 to the coordinate system for the space for which the staticimages are acquired by the wider field-of-view sensor 22. This processmay be repeated one or more additional times until the resulting erroris determined by the control system 28 to satisfy the predefined errorthreshold, thereby demonstrating convergence of the trained model.

Once the model has been trained, such as described above in conjunctionwith FIG. 4, the control system 28 is configured to utilize the model todetermine the 3D position and orientation of a fiducial marker and, inturn, an object, such as a robot manipulator, upon which the fiducialmarker is mounted. In some embodiments, a larger fiducial marker, suchas a larger ChaRuCo pattern, may be utilizing during the training phaseto increase the accuracy of the model than is used following thetraining phase during use of the model to determine the 3D position andorientation of the fiducial marker. As shown in block 70 of FIG. 7, thewider field-of-view sensor 24 of this example embodiment is configuredto acquire a static image of the space in which the fiducial marker isdisposed. The pan-tilt unit 26 is configured to support the narrowerfield-of-view sensor 24 and to controllably define the pan and tiltangles at which the narrower field-of-view is positioned relative to thefiducial marker such that at least a portion of the fiducial marker iswithin the field-of-view of the narrower field-of-view sensor. See block72. While at the pan and tilt angles as defined by the pan-tilt unit 26,the narrower field-of-view sensor 24 is configured to acquire an imageof at least a portion of the fiducial marker. See block 74. Unlike theplurality of images of at least a portion of fiducial marker acquired atdifferent pan and tilt angles during the training process, the narrowerfield-of-view sensor 24 of this example embodiment need only acquire asingle image of at least a portion of the fiducial marker, although moreimages may be acquired in other embodiments. Additionally, the controlsystem 28 need not be recalibrated after having moved the narrowerfield-of-view sensor 24, thereby improving the efficiency of theapparatus 20.

The control system 28 of this example embodiment is configured toutilize the transformation defined by the trained model to determine anestimate of the position and orientation of the fiducial marker and, inturn, the object that carries the fiducial marker in the coordinatesystem, e.g., the world coordinate system, for the space for which thestatic image is acquired by the wider fielded view sensor 22. See block76 of FIG. 7. This estimate of the position and orientation of thefiducial marker is based on the images acquired from the widerfield-of-view sensor 22 and the narrower field-of-view sensor 24 viewingthe fiducial marker and also based on the pan and tilt angles at whichthe narrower field-of-view sensor is positioned relative to the fiducialmarker upon acquiring the image. However, the transformation utilized bythe control system 28 is independent of any position information for thewider field-of-view sensor 22 and the narrower field-of-view sensor 24.Thus, the control system 28 need not know the position of the widerfield-of-view and narrower field-of-view sensors 22, 24 in order todetermine the position and orientation of the fiducial marker within theworld coordinate system.

In an example embodiment, the control system 28 is first configured todetect the pattern of the fiducial marker, such as by utilizing theOpenCV library of vision functions, and to then determine the positionand orientation of the fiducial marker in the local coordinate system ofthe narrower field-of-view sensor 24. In this example embodiment, thecontrol system 28 is then configured to utilize the transformation,based upon the trained model, to convert the position and orientation ofthe fiducial marker from the local coordinate system of the narrowerfield-of-view sensor 24 to the position and orientation to the fiducialmarker in the coordinate system, e.g., the world coordinate system, forthe space for which the static image is acquired by the widerfield-of-view sensor 22 based upon the pan and tilt angles of thepan-tilt unit 26.

Because of the narrower field-of-view in, some embodiments, the greaterresolution of the narrower field-of-view sensor 24 relative to the widerfield-of-view sensor 22, the narrower field-of-view sensor is configuredto acquire the image of at least a portion of fiducial marker with morepixels being representative of the fiducial marker than the static imageacquired by the wider field-of-view sensor. By placing more pixels onthe fiducial marker, the position and orientation of the fiducial markermay be more accurately and reliably determined by reference to the imageacquired by the narrower field-of-view sensor 24. Thus, the fusion ofthe information extracted from the images acquired by the widerfield-of-view sensor 22 and the narrower field-of-view sensor 24 allowsthe apparatus 20 to place more pixels on the fiducial marker with thenarrower field-of-view sensor while maintaining a constant, full view ofthe space with the wider field-of-view sensor. In addition, by utilizingthe trained model having model parameters that were determined tominimize a loss function defining an error metric between the positionand orientation of a fiducial marker in the world coordinate system ofthe wider field-of-view sensor 22 and in the local coordinate system ofthe narrower field-of-view sensor 24, the control system 28 of anexample embodiment is configured to determine the estimated position andorientation of the fiducial marker in real time with the acquisition bythe image of at least the portion of fiducial marker by the narrowerfield-of-view sensor. Thus, the transformation provided by the apparatus20 of this example embodiment is both computationally efficient andcapable of being performed in a timely manner.

Many modifications and other aspects of the disclosure set forth hereinwill come to mind to one skilled in the art to which this disclosurepertains having the benefit of the teachings presented in the foregoingdescriptions and the associated drawings. Therefore, it is to beunderstood that the disclosure is not to be limited to the specificaspects disclosed and that modifications and other aspects are intendedto be included within the scope of the appended claims. Althoughspecific terms are employed herein, they are used in a generic anddescriptive sense only and not for purposes of limitation.

That which is claimed:
 1. An apparatus for training a model to determinea three dimensional (3D) position and orientation of a fiducial marker,the apparatus comprising: a wider field-of-view sensor configured toacquire a static image of a space in which the fiducial marker isdisposed; a narrower field-of-view sensor configured to acquire aplurality of images of at least a portion of the fiducial marker; apan-tilt unit configured to support the narrower field-of-view sensorand to controllably alter a pan angle and a tilt angle at which thenarrower field-of-view sensor is positioned relative to the fiducialmarker such that the narrower field-of-view sensor is configured toacquire images of the fiducial marker at different pan and tilt angles;and a control system configured to determine a transformation ofposition and orientation information determined from the images acquiredby the narrower field-of-view sensor to a coordinate system for thespace for which the static image is acquired by the wider field-of-viewsensor.
 2. The apparatus of claim 1, wherein the control system isconfigured to determine the transformation by performing a trainingprocess comprising collecting the plurality of images from the narrowerfield-of-view sensor, processing the images in which the fiducial markeris identified in each image, and using the model to determine atransformation matrix for the transformation of the position andorientation information determined from the images acquired by thenarrower field-of-view sensor to the coordinate system for the space forwhich the static image is acquired by the wider field-of-view sensor. 3.The apparatus of claim 1, wherein the wider field-of-view sensor isconfigured to acquire a second static image of the space in which thefiducial marker is disposed after the fiducial marker has beenrepositioned, wherein the narrower field-of-view sensor is configured toacquire a second plurality of images of at least a portion of thefiducial marker at different combinations of pan and tilt angles afterthe fiducial marker has been repositioned, and wherein the controlsystem is further configured to determine the transformation of positionand orientation information determined from the second plurality ofimages acquired by the narrower field-of-view sensor to the coordinatesystem for the space for which the static images are acquired by thewider field-of-view sensor.
 4. The apparatus of claim 1, wherein thecontrol system is configured to determine the transformation by rotatinga position and orientation of the fiducial marker determined in a localcoordinate system of the narrower field-of-view sensor to a pancoordinate system and a tilt coordinate system defined by the pan-tiltunit.
 5. The apparatus of claim 4, wherein the control system is furtherconfigured to determine the transformation by translating the positionand orientation of the fiducial marker determined in the localcoordinate system of the narrower field-of-view sensor from an origin tothe pan coordinate system and the tilt coordinate system defined by thepan-tilt unit.
 6. The apparatus of claim 5, wherein the control systemis further configured to determine the transformation by transformingfrom the local coordinate system at pan and tilt angles of 0° to thecoordinate system for the space for which the static image is acquiredby the wider field-of-view sensor.
 7. The apparatus of claim 1, whereinthe control system is configured to determine the transformation bydetermining model parameters to minimize a loss function defining anerror metric between a position and orientation of the fiducial markerin the coordinate system for the space for which the static image isacquired by the wider field-of-view sensor and in a local coordinatesystem of the narrower field-of-view sensor.
 8. A method for training amodel to determine a three dimensional (3D) position and orientation ofa fiducial marker, the method comprising: acquiring, with a widerfield-of-view sensor, a static image of a space in which the fiducialmarker is disposed; sequentially positioning a narrower field-of-viewsensor to have different combinations of a pan angle and a tilt anglerelative to the fiducial marker; acquiring, with the narrowerfield-of-view sensor, a plurality of images of at least a portion of thefiducial marker at different pan and tilt angles; and determining atransformation of position and orientation information determined fromthe images acquired by the narrower field-of-view sensor to a coordinatesystem for the space for which the static image is acquired by the widerfield-of-view sensor.
 9. The method of claim 8, wherein determining thetransformation comprises performing a training process, and whereinperforming the training process comprises collecting the plurality ofimages from the narrower field-of-view sensor, processing the images inwhich the fiducial marker is identified in each image, and using themodel to determine a transformation matrix for the transformation of theposition and orientation information determined from the images acquiredby the narrower field-of-view sensor to the coordinate system for thespace for which the static image is acquired by the wider field-of-viewsensor.
 10. The method of claim 8, further comprising: acquiring, withthe wider field-of-view sensor, a second static image of the space inwhich the fiducial marker is disposed after the fiducial marker has beenrepositioned; acquiring, with the narrower field-of-view sensor, asecond plurality of images of at least a portion of the fiducial markerat different combinations of pan and tilt angles after the fiducialmarker has been repositioned; and determining the transformation ofposition and orientation information determined from the secondplurality of images acquired by the narrower field-of-view sensor to thecoordinate system for the space for which the static images are acquiredby the wider field-of-view sensor.
 11. The method of claim 8, whereindetermining the transformation comprises determining model parameters tominimize a loss function defining an error metric between a position andorientation of the fiducial marker in the coordinate system for thespace for which the static image is acquired by the wider field-of-viewsensor and in a local coordinate system of the narrower field-of-viewsensor.
 12. The method of claim 11, wherein determining the modelparameters comprises minimizing the loss function in a single step so asto determine all of the model parameters.
 13. The method of claim 11,wherein the model parameters comprise dynamic parameters and staticparameters, and wherein determining the model parameters comprises:determining the dynamic parameters and position and orientation of thefiducial marker in the coordinate system for the space for which thestatic image is acquired by the wider field-of-view sensor; anddetermining the static parameters while the dynamic parameters remainfixed.
 14. The method of claim 11, wherein determining the modelparameters comprises repeatedly minimizing disagreement between aplurality of six degree of freedom estimates of position and orientationof the fiducial marker from the coordinate system for the space fromwhich the static image is acquired by the wider field-of-view sensor andfrom the local coordinate system of the narrower field-of-view sensor byvarying one or more of the model parameters.
 15. The method of claim 11,wherein determining the model parameters comprises evaluating an errormetric between estimates of position and orientation from the widerfield-of-view sensor and from the narrower field-of-view sensor usingrespective positions of 3D features of one or more fiducial markers inrespective positions and orientations.
 16. An apparatus for determininga three dimensional (3D) position and orientation of a fiducial marker,the apparatus comprising: a wider field-of-view sensor configured toacquire a static image of a space in which the fiducial marker isdisposed; a narrower field-of-view sensor configured to acquire an imageof at least a portion of the fiducial marker; a pan-tilt unit configuredto support the narrower field-of-view sensor and to controllably definea pan angle and a tilt angle at which the narrower field-of-view sensoris positioned relative to the fiducial marker such that the narrowerfield-of-view sensor is configured to acquire the image of at least theportion of the fiducial marker; and a control system configured toutilize a transformation to determine, independent of any positioninformation for the wider field-of-view and narrower field-of-viewsensors, an estimate of the position and orientation of the fiducialmarker in a coordinate system for the space for which the static imageis acquired by the wider field-of-view sensor, based on the imagesacquired from the wider field-of-view and narrower field-of-view sensorsviewing the fiducial marker and also based on the pan and tilt angles atwhich the narrower field-of-view sensor is positioned relative to thefiducial marker upon acquiring the image.
 17. The apparatus of claim 16,wherein the control system is further configured to determine a positionand orientation of the fiducial marker in a local coordinate system ofthe narrower field-of-view sensor.
 18. The apparatus of claim 17,wherein the control system is configured to utilize the transformationto convert the position and orientation of the fiducial marker in thelocal coordinate system of the narrower field-of-view sensor to theposition and orientation of the fiducial marker in the coordinate systemfor the space for which the static image is acquired by the widerfield-of-view sensor based upon the pan angle and the tilt angle of thepan-tilt unit.
 19. The apparatus of claim 16, wherein the narrowerfield-of-view sensor is configured to acquire the image of at least theportion of the fiducial marker with more pixels representative of thefiducial marker than the static image acquired by the widerfield-of-view sensor.
 20. The apparatus of claim 16, wherein the controlsystem is configured to determine the estimate of the position andorientation of the fiducial marker in real time with acquisition of theimage of at least the portion of the fiducial marker by the narrowerfield-of-view sensor.